Methods and Apparatus for Flow-Based Batching and Processing

ABSTRACT

Techniques are provided for managing a user space protocol stack are disclosed herein. A nexus in a kernel space can receive a packet from a packet pool, and extract information from the packet to generate a flow key indicating a particular flow for the packet. The nexus can further look up the flow key in a flow table to determine whether there is an existing flow key stored in the flow table matching the flow key of the packet, and store the packet into a batch of packets of the existing flow when the existing flow key matches the flow key of the packet. When a release condition being met, the nexus can release the batch of packets of the existing flow to a user space protocol stack within a user space application through a channel communicatively coupled to the nexus and the user space protocol stack.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/038,520, filed Jun. 12, 2020, which is incorporated by referenceherein in its entirety.

This application is related to the subject matter of U.S. ProvisionalPatent Application Ser. No. 63/038,473 filed on Jun. 12, 2020 andentitled “Methods and Apparatus for Cross-Layer Transport Awareness,”which is incorporated herein by reference in its entirety.

This following applications are incorporated by reference: U.S. patentapplication Ser. No. 16/144,992 filed Sep. 27, 2018, now pending, andentitled “Methods and Apparatus for Single Entity Buffer PoolManagement”, U.S. patent application Ser. No. 16/146,533 filed Sep. 28,2018, now pending, and entitled “Methods and Apparatus for RegulatingNetworking Traffic in Bursty System Conditions”, U.S. patent applicationSer. No. 16/146,324 filed Sep. 28, 2018, now U.S. Pat. No. 10,978,224and entitled “Methods and Apparatus for Preventing Packet Spoofing withUser Space Communication Stacks”, U.S. patent application Ser. No.16/146,916 filed Sep. 28, 2018, now U.S. Pat. No. 10,819,831 andentitled “Methods and Apparatus for Channel Defunct Within User SpaceStack Architectures”, U.S. patent application Ser. No. 16/236,032 filedDec. 28, 2018, now pending and entitled “Methods and Apparatus forClassification of Flow Metadata with User Space Communication Stacks”,U.S. patent application Ser. No. 16/363,495 filed Mar. 25, 2019, nowpending and entitled “Methods and Apparatus for Dynamic Packet PoolConfiguration in Networking Stack Infrastructures”, U.S. patentapplication Ser. No. 16/365,462 filed on Mar. 26, 2019, now pending andentitled “Methods and Apparatus for Sharing and Arbitration of HostStack Information with User Space Communication Stacks”, U.S. patentapplication Ser. No. 16/368,338, now pending, filed on Mar. 28, 2019 andentitled “Methods and Apparatus for Memory Allocation and Reallocationin Networking Stack Infrastructures”, U.S. patent application Ser. No.16/365,484, filed on Mar. 26, 2019, now pending and entitled “Methodsand Apparatus for Virtualized Hardware Optimizations for User SpaceNetworking”, U.S. patent application Ser. No. 16/368,368 filed on Mar.28, 2019, now pending, and entitled “Methods and Apparatus for ActiveQueue Management in User Space Networking”, and U.S. patent applicationSer. No. 16/368,214 filed on Mar. 28, 2019, now pending and entitled“Methods and Apparatus for Self-Tuning Operation with User Space StackArchitectures”, U.S. Provisional Patent Application Ser. No. 62/906,617filed Sep. 26, 2019 and entitled “Methods and Apparatus for Low LatencyOperation in User Space Networking”, U.S. Provisional Patent ApplicationSer. No. 62/906,645 filed Sep. 26, 2019 and entitled “Methods andApparatus for Emerging Use Case Support in User Space Networking”, andU.S. Provisional Patent Application Ser. No. 62/906,657 filed Sep. 26,2019 and entitled “Methods and Apparatus for Device Driver Operation inNon-Kernel Space”, the contents of the foregoing being incorporatedherein by reference in their entireties.

COPYRIGHT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

The disclosure relates generally to the field of electronic devices, aswell as networks thereof. More particularly, the disclosure is directedto methods and apparatus for implementing computerized networking stackinfrastructures.

DESCRIPTION OF RELATED TECHNOLOGY

The consumer electronics industry has seen explosive growth in networkconnectivity; for example, Internet connectivity is now virtuallyubiquitous across many different device types for a variety of differentapplications and functionalities. The successful implementation ofnetwork connectivity over a myriad of different usage cases has beenenabled by, inter alia, the principles of modular design andabstraction. Specifically, the traditional network communicationparadigm incorporates multiple (generally) modular software “layers”into a “communication stack.” Each layer of the communication stackseparately manages its own implementation specific considerations andprovides an “abstracted” communication interface to the next layer. Inthis manner, different applications can communicate freely acrossdifferent devices without considering the underlying network transport.

Kernel space processing overhead is a function of packet volume withinexisting communication stacks. Specifically, each layer of thecommunication stack processes packets one-at-a-time, with little to noshared knowledge between layers. Unfortunately, modern user applicationsconsume a tremendous number of data packets, and the trend in consumerdata usage will only continue to increase. Furthermore, processing andmemory speeds appear to be asymptotically approaching the theoreticallimitations of existing semiconductor manufacturing. Thus, theconfluence of these factors (increasing overhead, aggressive consumerexpectations, and diminishing improvements in semiconductormanufacturing) present significant design challenges for consumerelectronics.

SUMMARY

The present disclosure satisfies the foregoing needs by providing, interalia, methods and apparatus for flow-based batching and processing.

In one aspect, methods and apparatus for providing in-kernel flowcontrol based on flow-batched data structures are disclosed. A methodfor managing a user space protocol stack can comprise receiving, by anexus in a kernel space, a packet from a packet pool, wherein the packetis associated with one or more flows, wherein the one or more flows areindividually identified by corresponding one or more flow keys;extracting information from the packet to generate a flow key of thepacket indicating a particular flow for the packet; looking up the flowkey in a flow table to determine whether there is an existing flow keystored in the flow table matching the flow key of the packet; storing,responsive to the existing flow key matching the flow key of the packet,the packet into a batch of packets of the existing flow, wherein thebatch of packets of the existing flow are stored in a flow-batched datastructure; and releasing, responsive to a release condition being met,the batch of packets of the existing flow to a user space protocol stackwithin a user space application through a channel communicativelycoupled to the nexus and the user space protocol stack

In one aspect, a non-transitory computer readable storage apparatusimplementing one or more of the aspects disclosed herein is disclosedand described. A non-transitory computer readable medium comprisingcomputer programs stored thereon, when executed by a processor, cancause a computerized apparatus to receive, by a nexus in a kernel space,a packet from a packet pool, wherein the packet is associated with oneor more flows, wherein the one or more flows are individually identifiedby corresponding one or more flow keys; extract information from thepacket to generate a flow key of the packet indicating a particular flowfor the packet; look up the flow key in a flow table to determinewhether there is an existing flow key stored in the flow table matchingthe flow key of the packet; store, responsive to the existing flow keymatching the flow key of the packet, the packet into a batch of packetsof the existing flow, wherein the batch of packets of the existing floware stored in a flow-batched data structure; and release, responsive toa release condition being met, the batch of packets of the existing flowto a user space protocol stack within a user space application through achannel communicatively coupled to the nexus and the user space protocolstack.

In one aspect, methods and apparatus for providing in-kernel flowcontrol based on flow-batched data structures are disclosed. A systemconfigured for managing packet flows, can include a storage device tostore a flow table; and a processor coupled to the storage device, andconfigured to operate a nexus in a kernel. The nexus is configured toreceive a packet from a packet pool, wherein the packet is associatedwith one or more flows, wherein the one or more flows are individuallyidentified by corresponding one or more flow keys; extract informationfrom the packet to generate a flow key of the packet indicating aparticular flow for the packet; look up the flow key in the flow tablein the storage to determine whether there is an existing flow key storedin the flow table matching the flow key of the packet; store, responsiveto the existing flow key matching the flow key of the packet, the packetinto a batch of packets of the existing flow, wherein the batch ofpackets of the existing flow are stored in a flow-batched datastructure; and release, responsive to a release condition being met, thebatch of packets of the existing flow to a user space protocol stackwithin a user space application through a channel communicativelycoupled to the nexus and the user space protocol stack.

In one aspect, methods and apparatus for representing multiple packetdata structures according to natural memory boundaries of an operatingsystem are disclosed.

In another aspect, computerized apparatus implementing one or more ofthe foregoing methods and/or apparatus are disclosed. In one embodiment,the computerized apparatus comprises a personal computer, such as adesktop computer.

In another aspect, the computerized apparatus comprises a mobile devicesuch as a smartphone or tablet computer, or laptop computer.

In yet another aspect, the computerized apparatus comprises a server orserver blade.

In still another aspect, the computerized apparatus comprises aGPU-based architecture optimized for deep learning or machine-learning.

In still another aspect, the computerized apparatus comprises a mediaplayer or rendering device, such as a portable player.

In still another aspect, the computerized apparatus comprises a hostserver or similar device, with hypervisor and configured to supportmultiple “guest” virtual machines (VMs) and/or containerizedapplications.

In another aspect, an integrated circuit (IC) device implementing one ormore of the foregoing aspects is disclosed and described. In oneembodiment, the IC device is embodied as a SoC (system on Chip) device.In another embodiment, an ASIC (application specific IC) is used as thebasis of the device. In yet another embodiment, a chip set (i.e.,multiple ICs used in coordinated fashion) is disclosed. In yet anotherembodiment, the device comprises a multi-logic block FPGA device. In afurther embodiment, the IC device is a modem or modem chipset. Inanother embodiment, the IC device comprises a multi-core processorarchitecture.

In another aspect, a computer readable storage apparatus implementingone or more of the foregoing aspects is disclosed and described. In oneembodiment, the computer readable apparatus comprises a program memory,or an EEPROM. In another embodiment, the apparatus includes a solidstate drive (SSD) or other mass storage device. In another embodiment,the apparatus comprises a USB or other “flash drive” or other suchportable removable storage device. In yet another embodiment, theapparatus comprises a “cloud” (network) based storage device which isremote from yet accessible via a computerized user or client electronicdevice. As another embodiment, the storage device is part of a server“farm” or “big data” compute environment.

In yet another aspect, an operating system (OS) is disclosed. In oneembodiment, the OS includes a user space and kernel space. In anotherembodiment, the OS is open-source (e.g., Linux)-based. In anotherembodiment, the OS is non-open (e.g., proprietary) based.

Other features and advantages of the present disclosure will immediatelybe recognized by persons of ordinary skill in the art with reference tothe attached drawings and detailed description of exemplary embodimentsas given below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a logical representation of a traditional network socket,useful for explaining various aspects of the present disclosure.

FIG. 2 is a logical representation of a computer system that implementsInput/Output (I/O) network control, useful for explaining variousaspects of the present disclosure.

FIG. 3 is a logical block diagram of one exemplary implementation ofTransport Layer Security (TLS), useful for explaining various aspects ofthe present disclosure.

FIG. 4 is a logical block diagram of an exemplary implementation of aVirtual Private Network (VPN), useful for explaining various aspects ofthe present disclosure.

FIG. 5 illustrates a logical block diagram of an exemplaryimplementation of application-based tuning, useful to explain variousother workload optimization complexities of emerging use cases.

FIG. 6 illustrates one logical representation of an exemplary user spacenetworking stack architecture, in accordance with the various aspects ofthe present disclosure.

FIG. 7 is a logical block diagram of an exemplary user space networkingstack, in accordance with the various aspects of the present disclosure.

FIG. 8 is a logical flow diagram of packet-based processing operationswithin a nexus, useful to illustrate various aspects of the presentdisclosure.

FIG. 9 is a logical block diagram of packet-based routing within a userspace networking stack architecture, useful to illustrate variousaspects of the present disclosure.

FIG. 10 is a graphical representation of memory allocations for packetdata structures, useful to explain various aspects of the presentdisclosure.

FIGS. 11A-11B are logical flow diagrams of flow-based processing withinan exemplary nexus, in accordance with the various aspects describedherein.

FIG. 12 is a logical block diagram of flow-based routing within anexemplary user space networking stack architecture, in accordance withthe various principles described herein.

FIG. 13 is a graphical representation of memory allocations forexemplary packet, super packet, and chained packet data structures, inaccordance with the various principles described herein.

FIG. 14 is an example computer system useful for implementing variousaspects described herein.

All figures ©Copyright 2017-2021 Apple Inc. All rights reserved.

DETAILED DESCRIPTION

Reference is now made to the drawings, wherein like numerals refer tolike parts throughout.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Existing Network SocketTechnologies

FIG. 1 illustrates one logical representation of a traditional networksocket 102, useful for explaining various aspects of the traditionalnetworking interface. A network “socket” is a virtualized internalnetwork endpoint for sending or receiving data at a single node in acomputer network. A network socket may be created (“opened”) ordestroyed (“closed”) and the manifest of network sockets may be storedas entries in a network resource table which may additionally includereference to various communication protocols (e.g., Transmission ControlProtocol (TCP) 104, User Datagram Protocol (UDP) 106, Inter-ProcessorCommunication (IPC) 108, etc.), destination, status, and any otheroperational processes (kernel extensions 112) and/or parameters); moregenerally, network sockets are a form of system resource.

As shown in FIG. 1, the socket 102 provides an application programminginterface (API) that spans between the user space and the kernel space.An API is a set of clearly defined methods of communication betweenvarious software components. An API specification commonly includes,without limitation: routines, data structures, object classes,variables, remote calls and/or any number of other software constructscommonly defined within the computing arts.

As a brief aside, user space is a portion of system memory that aprocessor executes user processes from. User space is relatively freelyand dynamically allocated for application software and a few devicedrivers. The kernel space is a portion of memory that a processorexecutes the kernel from. Kernel space is strictly reserved (usuallyduring the processor boot sequence) for running privileged operatingsystem (O/S) processes, extensions, and most device drivers. Forexample, each user space process normally runs in a specific memoryspace (its own “sandbox”) and cannot access the memory of otherprocesses unless explicitly allowed. In contrast, the kernel is the coreof a computer's operating system; the kernel can exert complete controlover all other processes in the system.

The term “operating system” may refer to software that controls andmanages access to hardware. An O/S commonly supports processingfunctions such as e.g., task scheduling, application execution, inputand output management, memory management, security, and peripheralaccess. As used herein, the term “application” refers to software thatcan interact with the hardware only via procedures and interfacesoffered by the O/S.

The term “privilege” may refer to any access restriction or permissionwhich restricts or permits processor execution. System privileges arecommonly used within the computing arts to, inter alia, mitigate thepotential damage of a computer security vulnerability. For instance, aproperly privileged computer system will prevent malicious softwareapplications from affecting data and task execution associated withother applications and the kernel.

As used herein, the term “in-kernel” and/or “kernel space” may refer todata and/or processes that are stored in, and/or have privilege toaccess the kernel space memory allocations. In contrast, the terms“non-kernel” and/or “user space” refers to data and/or processes thatare not privileged to access the kernel space memory allocations. Inparticular, user space represents the address space specific to the userprocess, whereas non-kernel space represents address space which is notin-kernel, but which may or may not be specific to user processes.

As previously noted, the illustrated socket 102 provides access toTransmission Control Protocol (TCP) 104, User Datagram Protocol (UDP)106, and Inter-Processor Communication (IPC) 108. TCP, UDP, and IPC arevarious suites of transmission protocols each offering differentcapabilities and/or functionalities. For example, UDP is a minimalmessage-oriented encapsulation protocol that provides no guarantees tothe upper layer protocol for message delivery and the UDP layer retainsno state of UDP messages once sent. UDP is commonly used for real-time,interactive applications (e.g., video chat, voice over IP (VoIP) whereloss of packets is acceptable. In contrast, TCP provides reliable,ordered, and error-checked delivery of data via a retransmission andacknowledgement scheme; TCP is generally used for file transfers wherepacket loss is unacceptable, and transmission latency is flexible.

As used herein, the term “encapsulation protocol” may refer to modularcommunication protocols in which logically separate functions in thenetwork are abstracted from their underlying structures by inclusion orinformation hiding within higher level objects. For example, in oneexemplary embodiment, UDP provides extra information (ports numbering).

As used herein, the term “transport protocol” may refer to communicationprotocols that transport data between logical endpoints. A transportprotocol may include encapsulation protocol functionality.

Both TCP and UDP are commonly layered over an Internet Protocol (IP) 110for transmission. IP is a connectionless protocol for use onpacket-switched networks that provides a “best effort delivery”. Besteffort delivery does not guarantee delivery, nor does it assure propersequencing or avoidance of duplicate delivery. Generally these aspectsare addressed by TCP or another transport protocol based on UDP.

As a brief aside, consider a web browser that opens a webpage; the webbrowser application would generally open a number of network sockets todownload and/or interact with the various digital assets of the webpage(e.g., for a relatively common place webpage, this could entailinstantiating ˜300 sockets). The web browser can write (or read) data tothe socket; thereafter, the socket object executes system calls withinkernel space to copy (or fetch) data to data structures in the kernelspace.

As used herein, the term “domain” may refer to a self-contained memoryallocation e.g., user space, kernel space. A “domain crossing” may referto a transaction, event, or process that “crosses” from one domain toanother domain. For example, writing to a network socket from the userspace to the kernel space constitutes a domain crossing access.

In the context of a Berkeley Software Distribution (BSD) basednetworking implementation, data that is transacted within the kernelspace is stored in memory buffers that are also commonly referred to as“mbufs”. Each mbuf is a fixed size memory buffer that is usedgenerically for transfers (mbufs are used regardless of the callingprocess e.g., TCP, UDP, etc.). Arbitrarily sized data can be split intomultiple mbufs and retrieved one at a time or (depending on systemsupport) retrieved using “scatter-gather” direct memory access (DMA)(“scatter-gather” refers to the process of gathering data from, orscattering data into, a given set of buffers). Each mbuf transfer isparameterized by a single identified mbuf.

Notably, each socket transfer can create multiple mbuf transfers, whereeach mbuf transfer copies (or fetches) data from a single mbuf at atime. As a further complication, because the socket spans both: (i) userspace (limited privileges) and (ii) kernel space (privileged withoutlimitation), the socket transfer verifies that each mbuf copy into/outof kernel space is valid. More directly, the verification processensures that the data access is not malicious, corrupted, and/ormalformed (i.e., that the transfer is appropriately sized and is to/froman appropriate area).

The processing overhead associated with domain crossing is a non-trivialprocessing cost. Processing cost affects user experience both directlyand indirectly. A processor has a fixed amount of processing cyclesevery second; thus cycles that are used for transfer verificationdetract from more user perceptible tasks (e.g., rendering a video oraudio stream). Additionally, processor activity consumes power; thus,increases in processing overhead increases power consumption.

Referring back to FIG. 1, in addition to the generic TCP 104, UDP 106,and IPC 108 communication suites, the illustrated socket 102 also mayprovide access to various kernel extensions 112. A kernel extension is adynamically loaded bundle of executable code that executes from kernelspace. Kernel extensions may be used to perform low-level tasks thatcannot be performed in user space. These low-level tasks typically fallinto one or more of: low-level device drivers, network filters, and/orfile systems. Examples of sockets and/or extensions include withoutlimitation: route (IP route handling), ndrv (packet 802.1X handling),key (key management), unix (translations for Unix systems), kernelcontrol, kernel events, parental controls, intrusion detection, contentfiltering, hypervisors, and/or any number of other kernel tasking.

Kernel extensions and public APIs enable, for example, 3^(rd) partysoftware developers to develop a wide variety of applications that caninteract with a computer system at even the lowest layers ofabstraction. For example, kernel extensions can enable socket levelfiltering, IP level filtering, and even device interface filtering. Inthe current consumer applications space, many emerging technologies nowrely on closely coupled interfaces to the hardware and kernelfunctionality. For example, many security applications “sniff” networktraffic to detect malicious traffic or filter undesirable content; thisrequires access to other application sandboxes (a level of privilegethat is normally reserved for the kernel).

Unfortunately, 3^(rd) party kernel extensions can be dangerous and/orundesirable. As previously noted, software applications are restrictedfor security and stability reasons; however the kernel is largelyunrestricted. A 3^(rd) party kernel extension can introduce instabilityissues because the 3rd party kernel extensions run in the same addressspace as the kernel itself (which is outside the purview of traditionalmemory read/write protections based on memory allocations). Illegalmemory accesses can result in segmentation faults and memorycorruptions. Furthermore, unsecure kernel extension can create securityvulnerabilities that can be exploited by malware. Additionally, evenwhere correctly used, a kernel extension can expose a user's data to the3^(rd) party software developer. This heightened level of access mayraise privacy concerns (e.g., the 3^(rd) party developer may have accessto browsing habits, etc.).

Exiting Performance Optimization Technologies

FIG. 2 illustrates one logical representation of a computer system thatimplements Input/Output (I/O) network control, useful for explainingvarious aspects of traditional network optimization. As depictedtherein, a software application 202 executing from user space opensmultiple sockets 204 to communicate with e.g., a web server. Each of thesockets interfaces with a Data Link Interface Layer (DLIL) 206.

The DLIL 206 provides a common interface layer to each of the variousphysical device drivers which will handle the subsequent data transfer(e.g., Ethernet, Wi-Fi, cellular, etc.). The DLIL performs a number ofsystem-wide holistic network traffic management functions. In one suchimplementation, the DLIL is responsible for BSD Virtual Interfaces,IOKit Interfaces (e.g., DLIL is the entity by which IOKit based networkdrivers are connected to the networking stack), Active Queue Management(AQM), flow control and advisory action, etc. In most cases, the devicedriver 208 may be handled by an external device (e.g., a basebandco-processor), thus the DLIL 206 is usually (but not always) the lowestlayer of the network communication stack.

During normal operation, the computer system will logically segment itstasks to optimize overall system operation. In particular, a processorwill execute a task, and then “context switch” to another task, therebyensuring that any single process thread does not monopolize processorresources from start to finish. More directly, a context switch is theprocess of storing the state of a process, or of a thread, so that itcan be restored and execution resumed from the same point later. Thisallows multiple processes to share a single processor. However,excessive amounts of context switching can slow processor performancedown. Notably, while the present discussion is primarily discussedwithin the context of a single processor for ease of understanding,multi-processor systems have analogous concepts (e.g., multipleprocessors also perform context switching, although contexts may notnecessarily be resumed by the same processor).

For example, consider the following example of a packet reception.Packets arrive at the device driver 208A. The hardware managed by thedevice driver 208A may notify the processor via e.g., a doorbell signal(e.g., an interrupt). The device driver 208A work loop thread handlesthe hardware interrupt/doorbell, then signals the DLIL thread (Loop 1210). The processor services the device driver 208A with high priority,thereby ensuring that the device driver 208A operation is notbottlenecked (e.g., that the data does not overflow the device driver'smemory and/or that the device driver does not stall). Once the data hasbeen moved out of the device driver, the processor can context switch toother tasks.

At a later point, the processor can pick up the DLIL 206 executionprocess again. The processor determines which socket the packets shouldbe routed to (e.g., socket 204A) and routes the packet dataappropriately (Loop 2 212). During this loop, the DLIL thread takes eachpacket, and moves each one sequentially into the socket memory space.Again, the processor can context switch to other tasks so as to ensurethat the DLIL task does not block other concurrently executedprocessing.

Subsequently thereafter, when the socket has the complete packet datatransfer the processor can wake the user space application and deliverthe packet into user space memory (Loop 3 214). Generally, user spaceapplications are treated at lower priority than kernel tasks; this canbe reflected by larger time intervals between suspension and resumption.While the foregoing discussion is presented in the context of packetreception, artisans of ordinary skill in the related arts will readilyappreciate, given the contents of the present disclosure, that theprocess is substantially reversed for packet transmission.

As demonstrated in the foregoing example, context switching ensures thattasks of different processing priority are allocated commensurateamounts of processing time. For example, a processor can spendsignificantly more time executing tasks of relatively high priority, andservice lower priority tasks on an as-needed basis. As a brief aside,human perception is much more forgiving than hardware operation.Consequently, kernel tasks are generally performed at a much higherpriority than user space applications. The difference in prioritybetween kernel and user space allows the kernel to handle immediatesystem management (e.g., hardware interrupts, and queue overflow) in atimely manner, with minimal noticeable impact to the user experience.

Moreover, FIG. 2 is substantially representative of every implementationof the traditional network communications stack. While implementationsmay vary from this illustrative example, virtually all networking stacksshare substantially the same delivery mechanism. The traditional networkcommunications stack schema (such as the BSD architecture andderivatives therefrom) have been very popular for the past 30 years dueto its relative stability of implementation and versatility across manydifferent device platforms. For example, the Assignee hereof hasdeveloped and implemented the same networking stack across virtually allof its products (e.g., MacBook®, iMac®, iPad®, and iPhone®, AppleWatch®, etc.).

Unfortunately, changing tastes in consumer expectations cannot beeffectively addressed with the one-size-fits-all model and theconservative in-kernel traditional networking stack. Artisans ofordinary skill in the related arts will readily appreciate, given thecontents of the present disclosure, that different device platforms havedifferent capabilities; for example, a desktop processor hassignificantly more processing and memory capability than a mobile phoneprocessor. More directly, the “one-size-fits-all” solution does notaccount for the underlying platform capabilities and/or applicationrequirements, and thus is not optimized for performance. Fine-tuning thetraditional networking stack for performance based on various “tailored”special cases results in an inordinate amount of software complexitywhich is untenable to support across the entire ecosystem of devices.

Emerging Use Cases

FIG. 3 illustrates a logical block diagram of one exemplaryimplementation of Transport Layer Security (TLS) (the successor toSecure Sockets Layer (SSL)), useful to explain user/kernel spaceintegration complexities of emerging use cases.

As shown, an application executing from user space can open a HypertextTransfer Protocol (HTTP) session 302 with a TLS security layer 304 inorder to securely transfer data (Application Transport Security (ATS)services) over a network socket 306 that offers TCP/IP transport 308,310.

As a brief aside, TLS is a record based protocol; in other words, TLSuses data records which are arbitrarily sized (e.g., up to 16kilobytes). In contrast, TCP is a byte stream protocol (i.e., a byte hasa fixed length of eight (8) bits). Consequently, the TCP layersubdivides TLS records into a sequentially ordered set of bytes fordelivery. The receiver of the TCP byte stream reconstructs TLS recordsfrom the TCP byte stream by receiving each TCP packet, re-ordering thepackets according to sequential numbering to recreate the byte streamand extracting the TLS record from the aggregated byte stream. Notably,every TCP packet of the sequence must be present before the TLS recordcan be reconstructed. Even though TCP can provide reliable deliveryunder lossy network conditions, there are a number of situations whereTLS record delivery could fail. For example, under ideal conditions TCPisolates packet loss from its client (TLS in this example), and a singleTCP packet loss should not result in failed TLS record delivery.However, the TLS layer or the application above may incorporate atimeout strategy in a manner that is unaware of the underlying TCPconditions. Thus, if there's significant packet loss in the network, theTLS timeout may be hit (and thus result in a failure to the application)even though TCP would normally provide reliable delivery.

Referring back to FIG. 3, virtually every modern operating systemexecutes TLS from user space when e.g., securely connecting to othernetwork entities, inter alia, a web browser instance and a server. Butexisting implementations of TLS are not executed from the kernel (orother privileged software layer) due to e.g., the complexity of errorhandling within the kernel. However, as a practical matter, TLS wouldoperate significantly better with information regarding the currentnetworking conditions (held in the kernel).

Ideally, the TLS layer should set TLS record sizes based on networkcondition information. In particular, large TLS records can efficientlyuse network bandwidth, but require many successful TCP packetdeliveries. In contrast, small TLS records incur significantly morenetwork overhead, but can survive poor bandwidth conditions.Unfortunately, networking condition information is lower layerinformation that is available to the kernel space (e.g., the DLIL anddevice drivers), but generally restricted from user space applications.Some 3^(rd) party application developers and device manufacturers haveincorporated kernel extensions (or similar operating systemcapabilities) to provide network condition information to the TLS userspace applications; however, kernel extensions are undesirable due tothe aforementioned security and privacy concerns. Alternately, some3^(rd) party applications infer the presence of lossy network conditionsbased on historic TLS record loss. Such inferences are an indirectmeasure and significantly less accurate and lag behind real-timeinformation (i.e., previous packet loss often does not predict futurepacket loss).

FIG. 4 illustrates a logical block diagram of an exemplaryimplementation of a Virtual Private Network (VPN), useful to explainrecursive/cross-layer protocol layer complexities of emerging use cases.

As shown, an application executing from user space can open a VirtualPrivate Network (VPN) session 402 over a network socket 406 that offersTCP/IP transport 408, 410. The VPN session is secured with EncapsulatingSecurity Protocol (ESP) 412. The encrypted packet is securely tunneledvia TLS 404 (in user space) and recursively sent again over TCP/IPtransport 408, 410.

As illustrated within FIG. 4, the exemplary VPN tunnel starts in userspace, crosses into kernel space, returns back to user space, and thencrosses back into kernel space before being transferred. Each of thedomain crossings results in costly context switches and data shufflingboth of which are processor intensive and inefficient. More directly,every time data traverses from user space to kernel space, the data mustbe validated (which takes non-trivial processing time). Additionally,context switching can introduce significant latency while the task issuspended.

Artisans of ordinary skill in the related arts, given the contents ofthe present disclosure, will readily appreciate that the exemplaryrecursive cross layer transaction of FIG. 4 is merely illustrative of abroad range of applications which use increasingly exotic protocol layercompositions. For example, applications that traverse the applicationproxy/agent data path commonly require tunneling TCP (kernel space) overapplication proxy/agent data path (user space) over UDP/IP (kernelspace). Another common implementation is IP (kernel space) over QuickUDP Internet Connections (QUIC) (user space) over UDP/IP (kernel space).

FIG. 5 illustrates a logical block diagram of an exemplaryimplementation of application-based tuning, useful to explain variousother workload optimization complexities of emerging use cases.

As shown, three (3) different concurrently executed applications (e.g.,a real time application 502, interactive application 504, and filetransfer applications 506) in user space, each open a session overnetwork sockets 508 (508A, 508B, 508C) that offer TCP/UDP/IP transport510/512. Depending on the type of physical interface required, thesessions are switched to BSD network interfaces (ifnet) 514 (514A, 514B,514C) which handle the appropriate technology. Three differentillustrated technology drivers are shown: Wi-Fi 516, Bluetooth 518, andcellular 520.

It is well understood within the networking arts that differentapplication types are associated with different capabilities andrequirements. One such example is real time applications 502, commonlyused for e.g., streaming audio/visual and/or other “live” data. Realtime data has significant latency and/or throughput restrictions;moreover, certain real time applications may not require (and/orsupport) retransmission for reliable delivery of lost or corrupted data.Instead, real time applications may lower bandwidth requirements tocompensate for poor transmission quality (resulting in lower quality,but timely, delivered data).

Another such example is interactive applications 504, commonly used fore.g., human input/output. Interactive data should be delivered atlatencies that are below the human perceptible threshold (within severalmilliseconds) to ensure that the human experience is relativelyseamless. This latency interval may be long enough for a retransmission,depending on the underlying physical technology. Additionally, humanperception can be more or less tolerant of certain types of datacorruptions; for example, audio delays below 20 ms are generallyimperceptible, whereas audio corruptions (pops and clicks) arenoticeable. Consequently, some interactive applications may allow forsome level of error correction and/or adopt less aggressive bandwidthmanagement mechanisms depending on the acceptable performancerequirements for human perception.

In contrast to real time applications and interactive applications, filetransfer applications 506 require perfect data fidelity without latencyrestrictions. To these ends, most file transfer technologies supportretransmission of lost or corrupted data, and retransmission can haverelatively long attempt intervals (e.g., on the order of multipleseconds to a minute).

Similarly, within the communication arts, different communicationtechnologies are associated with different capabilities andrequirements. For example, Wi-Fi 516 (wireless local area networkingbased on IEEE 802.11) is heavily based on contention-based access and isbest suited for high bandwidth deliveries with reasonable latency. Wi-Fiis commonly used for file transfer type applications. Bluetooth 518(personal area networking) is commonly used for low data rate and lowlatency applications. Bluetooth is commonly used for human interfacedevices (e.g., headphones, keyboards, and mice). Cellular networktechnologies 520 often provide non-contention-based access (e.g.,dedicated user access) and can be used over varying geographic ranges.Cellular voice or video delivery is a good example of streaming dataapplications. Artisans of ordinary skill in the related arts willreadily recognize that the foregoing examples are purely illustrative,and that different communication technologies are often used to supporta variety of different types of application data. For example, Wi-Fi 516can support file transfer, real time data transmission and/orinteractive data with equivalent success.

Referring back to FIG. 5, the presence of multiple concurrentlyexecuting applications of FIG. 5 (real time application 502, interactiveapplication 504, and file transfer applications 506) illustrates thecomplexities of multi-threaded operation. As shown therein, theexemplary multi-threaded operation incurs a number of server loops. Eachserver loop represents a logical break in the process during which theprocessor can context switch (see also aforementioned discussion ofExisting Performance Optimization Technologies, and corresponding FIG.2).

Moreover, in the computing arts, a “locking” synchronization mechanismis used by the kernel to enforce access limits (e.g., mutual exclusion)on resources in multi-threaded execution. During operation, each threadacquires a lock before accessing the corresponding locked resourcesdata. In other words, at any point in time, the processor is necessarilylimited to only the resources available to its currently executingprocess thread.

Unfortunately, each of the applications has different latency,throughput and processing utilization requirements. Since, each of thenetwork interfaces is sending and receiving data at different times, indifferent amounts, and with different levels of priority. From a purelylogistical standpoint, the kernel is constantly juggling between highpriority kernel threads (to ensure that the high priority hardwareactivities do not stall out) while still servicing each of itsconcurrently running applications to attempt to provide acceptablelevels of service. In some cases, however, the kernel is bottlenecked bythe processor's capabilities. Under such situations, some threads willbe deprioritized; currently, the traditional networking stackarchitecture is unable it clearly identify which threads can bedeprioritized while still providing acceptable user service.

For example, consider an “expected use” device of FIG. 5; the processoris designed for the expected use case of providing streaming video.Designing for expected use cases allows the device manufacturer to useless capable, but adequate components thereby reducing bill of materials(BOM) costs and/or offering features at a reasonable price point forconsumers. In this case, a processor is selected that nominally meetsthe requirements for a streaming video application that is receivingstreaming video data via one of the network interfaces (e.g., the Wi-Fiinterface), and constantly servicing the kernel threads associated withit. Rendering the video with a real time application 502 from thereceived data is a user space application that is executed concurrentlybut at a significantly lower priority. During expected usage, the videorendering is adequate.

Unfortunately, the addition of an unexpected amount of additionalsecondary interactive applications 504 (e.g., remote control interface,headphones, and/or other interface devices) and/or background filetransfer applications can easily overwhelm the processor. Specifically,the primary real time application does not get enough CPU cycles to runwithin its time budget, because the kernel threads handling networkingare selected at a higher priority. In other words, the user spaceapplication is not able to depress the priority of kernel networkingthreads (which are servicing both the primary and secondary processes).This can result in significantly worse user experience when the videorendering stalls out (video frame misses or video frame drops); whereassimply slowing down a file transfer or degrading the interactioninterface may have been preferable.

Prior art solutions have tailored software for specific deviceimplementations (e.g., the Apple TV®). For example, the device can bespecifically programmed for an expected use. However, tailored solutionsare becoming increasingly common and by extension the exceptions haveswallowed the more generic use case. Moreover, tailored solutions areundesirable from multiple software maintenance standpoints. Devices havelimited productive lifetimes, and software upkeep is non-trivial.

Ideally, a per-application or per-profile workload optimization wouldenable a single processor (or multiple processors) to intelligentlydetermine when and/or how too intelligently context switch and/orprioritize its application load (e.g., in the example of FIG. 5, toprioritize video decode). Unfortunately, such solutions are not feasiblewithin the context of the existing generic network sockets and genericnetwork interfaces to a monolithic communications stack.

Exemplary User Space Networking Architecture

A networking stack architecture and technology that caters to the needsof non-kernel-based networking use cases is disclosed herein. Unlikeprior art monolithic networking stacks, the exemplary networking stackarchitecture described hereinafter includes various components that spanmultiple domains (both in-kernel, and non-kernel), with varyingtransport compositions, workload characteristics and parameters.

The user space networking stack architecture provides an efficientinfrastructure to transfer data across domains (user space, non-kernel,and kernel). Unlike the traditional networking paradigm that hides theunderlying networking tasks within the kernel and substantially limitscontrol thereof by any non-kernel applications, the various embodimentsdescribed herein enable faster and more efficient cross domain datatransfers.

Various embodiments of the present disclosure provide a faster and moreefficient packet input/output (I/O) infrastructure than prior arttechniques. Specifically, unlike traditional networking stacks that usea “socket” based communication, disclosed embodiments can transfer datadirectly between the kernel and user space domains. Direct transferreduces the per-byte and per-packet costs relative to socket-basedcommunication. Additionally, direct transfer can improve observabilityand accountability with traffic monitoring.

FIG. 6 illustrates one logical representation of an exemplary user spacenetworking stack architecture, in accordance with the various aspects ofthe present disclosure. While the system depicts a plurality of userspace applications 602 and/or legacy applications 612, artisans ofordinary skill will readily appreciate given the contents of presentdisclosure that the disclosed embodiments may be used within singleapplication systems with equivalent success.

As shown, a user space application 602 can initiate a network connectionby instancing user space protocol stacks 604. Each user space protocolstacks includes network extensions for e.g., TCP/UDP/QUIC/IP,cryptography, framing, multiplexing, tunneling, and/or any number ofother networking stack functionalities. Each user space protocol stack604 communicates with one or more nexuses 608 via a channel input/output(I/O) 606. Each nexus 608 manages access to the network drivers 610.Additionally, shown is legacy application 612 support via existingnetwork socket technologies 614. While the illustrated embodiment showsnexus connections to both user space and in-kernel networking stacks, itis appreciated that the nexus may also enable e.g., non-kernelnetworking stacks (such as may be used by a daemon or other non-kernel,non-user process).

The following topical sections hereinafter describe the salient featuresof the various logical constructs in greater detail.

Exemplary User Space I/O Infrastructure

In one embodiment, the non-kernel networking stack provides a directchannel input output (I/O) 606. In one such implementation, the channelI/O 606 is included as part of the user space protocol stack 604. Moredirectly, the channel I/O 606 enables the delivery of packets as a rawdata I/O into kernel space with a single validation (e.g., only when theuser stack provides the data to the one or more nexuses 608). The datacan be directly accessed and/or manipulated in situ, the data need notbe copied to an intermediary buffer.

In one exemplary implementation, a channel is an I/O scheme leveragingkernel-managed shared memory. During an access, the channel I/O ispresented to the process (e.g., the user process or kernel process) as afile descriptor-based object, rather than as data. In order to accessthe data, the process de-references the file descriptor for directaccess to the shared memory within kernel space. In one suchimplementation, the file descriptor-based object based I/O is compatiblewith existing operating system signaling and “eventing” (eventnotification/response) mechanisms. In one exemplary variant, the channelI/O is based on Inter Process Communication (IPC) packets.

As used herein, the term “descriptor” may refer to data structures thatindicate how other data is stored. Descriptors generally includemultiple parameters and can be used to identify more complex datastructures; for example, a descriptor may include one or more of type,size, address, tag, flag, headers, footers, metadata, structural linksto other data descriptors or locations, and/or any other number offormat or construction information.

Within the context of the present disclosure, as used herein, the term“pointer” may refer to a specific reference data type that “points” or“references” a location of data in memory. Typically, a pointer stores amemory address that is interpreted by a compiler as an absolute locationin system memory or a relative location in system memory based on e.g.,a base address, reference address, memory window, or other memorysubset. During operation, a pointer is “de-referenced” to recover thedata that is stored in the location of memory.

As used herein, the term “metadata” refers to data that describes data.Metadata varies widely in application, but generally falls into one ofthe descriptive, structural, and/or administrative categories.Descriptive metadata describes data in a manner to enable e.g.,discovery and/or identification. Common examples include withoutlimitation e.g., type, size, index tags, and keywords. Structuralmetadata describes the structure of the data e.g., how compound objectsare put together. Common examples include without limitation e.g.,prefix, postfix, table of contents, order, and/or any other informationthat describes the relationships and other characteristics of digitalmaterials. Administrative metadata provides information to help manage aresource; common examples include e.g., authorship and creationinformation, access privileges, and/or error checking and security-basedinformation (e.g., cyclic redundancy checks (CRC), parity, etc.).

In one embodiment, the channel I/O can be further leveraged to providedirect monitoring of its corresponding associated memory. More directly,unlike existing data transfers which are based on mbuf baseddivide/copy/move, etc., the channel I/O can provide (with appropriateviewing privileges) a direct window into the memory accesses of thesystem. Such implementations further simplify software development asdebugging and/or traffic monitoring can be performed directly ontraffic. Direct traffic monitoring can reduce errors attributed to falsepositives/false negatives caused by e.g., different software versioning,task scheduling, compiler settings, and/or other software introducedinaccuracies.

In one embodiment, the in-kernel network device drivers (e.g. Wi-Fi,Cellular, Ethernet) use simplified data movement models based on theaforementioned channel I/O scheme. More directly, the user spacenetworking stacks can directly interface to each of the variousdifferent technology-based network drivers via channel I/O; in thismanner, the user space networking stacks do not incur the traditionaldata mbuf based divide/copy/move penalties. Additionally, user spaceapplications can directly access user space networking components forimmediate traffic handling and processing.

Exemplary Nexus

In one embodiment, the networking stack connects to one or more nexus608. In one such implementation, the nexus 608 is a kernel space processthat arbitrates access to system resources including, without limitatione.g., shared memory within kernel space, network drivers, and/or otherkernel or user processes. In one such variant, the nexus 608 aggregatesone or more channels 606 together for access to the network drivers 610and/or shared kernel space memory.

In one exemplary implementation, a nexus is a kernel process thatdetermines the format and/or parameters of the data flowing through itsconnected channels. In some variants, the nexus may further performingress and/or egress filtering.

The nexus may use the determined format and/or parameter information tofacilitate one-to-one and one-to-many topologies. For example, the nexuscan create user-pipes for process-to-process channels; kernel-pipes forprocess-to-kernel channels; network interfaces for direct channelconnection from a process to in-kernel network drivers, or legacynetworking stack interfaces; and/or flow-switches for multiplexing flowsacross channels (e.g., switching a flow from one channel to one or moreother channels).

Additionally, in some variants the nexus may provide the format,parameter, and/or ingress egress information to kernel processes and/orone or more appropriately privileged user space processes.

In one embodiment, the nexus 608 may additionally ensure that there isfairness and/or appropriately prioritize each of its connected stacks.For example, within the context of FIG. 6, the nexus 608 balances thenetwork priorities of both the existing user space applicationnetworking stacks 604, as well as providing fair access for legacysocket-based access 614. For example, as previously alluded to, existingnetworking stacks could starve user space applications because thekernel threads handling the legacy networking stack operated at higherpriorities than user space applications. However, the exemplary nexus608 ensures that legacy applications do not monopolize system resourcesby appropriately servicing the user space network stacks as well as thelegacy network stack.

In one such embodiment, in-kernel, non-kernel, and/or user spaceinfrastructures ensure fairness and can reduce latency due to e.g.,buffer bloat (across channels in a given nexus, as well as flows withina channel). In other words, the in-kernel and/or user spaceinfrastructures can negotiate proper buffering sizes based on theexpected amount of traffic and/or network capabilities for each flow. Bybuffering data according to traffic and/or network capability, buffersare not undersized or oversized.

As a brief aside, “buffer bloat” is commonly used to describe e.g., highlatency caused by excessive buffering of packets. Specifically, bufferbloat may occur when excessively large buffers are used to support areal time streaming application. As a brief aside, TCP retransmissionmechanism relies on measuring the occurrence of packet drops todetermine the available bandwidth. Under certain congestion conditions,excessively large buffers can prevent the TCP feedback mechanism fromcorrectly inferring the presence of a network congestion event in atimely manner (the buffered packets “hide” the congestion, since theyare not dropped). Consequently, the buffers have to drain before TCPcongestion control resets and the TCP connection can correct itself.

Referring back to FIG. 6, in one embodiment, Active Queue Management(AQM) can be implemented in the kernel across one or more (potentiallyall) of the flow-switch clients (user space and in-kernel networkingstack instances). AQM refers to the intelligent culling of networkpackets associated with a network interface, to reduce networkcongestion. By dropping packets before the queue is full, the AQMensures no single buffer approaches its maximum size, and TCP feedbackmechanisms remain timely (thereby avoiding the aforementioned bufferbloat issues).

While the foregoing example is based on “fairness” standard, artisans ofordinary skill in the related arts will readily appreciate that otherschemes may be substituted with equivalent success given the contents ofthe present disclosure. For example, some embodiments may dynamically orstatically service the user application networking space with greater orless weight compared to the legacy socket-based access. For example,user application networking space may be more heavily weighted toimprove overall performance or functionality, whereas legacysocket-based access may be preferred where legacy applications arepreferentially supported.

Exemplary Network Extensions

In one embodiment of the present disclosure, a network extension isdisclosed. A network extension is an agent-based extension that istightly coupled to network control policies. The agent is executed bythe kernel and exposes libraries of network control functionality touser space applications. During operation, user space software canaccess kernel space functionality through the context and privileges ofthe agent.

As used herein, the term “agent” may refer to a software agent that actsfor a user space application or other program in a relationship ofagency with appropriate privileges. The agency relationship between theagent and the user space application implies the authority to decidewhich, if any, action is appropriate given the user application andkernel privileges. A software agent is privileged to negotiate with thekernel and other software agents regarding without limitation e.g.,scheduling, priority, collaboration, visibility, and/other sharing ofuser space and kernel space information. While the agent negotiates withthe kernel on behalf of the application, the kernel ultimately decideson scheduling, priority, etc.

Various benefits and efficiencies can be gained through the use ofnetwork extensions. In particular, user space applications can controlthe protocol stack down to the resolution of exposed threads (i.e., thethreads that are made available by the agent). In other words, softwareagents expose specific access to lower layer network functionality whichwas previously hidden or abstracted away from user space applications.For example, consider the previous examples of TLS record sizing (seee.g., FIG. 3, and related discussion); by exposing TCP networkconditions to the TLS application within the user space, the TLSapplication can correctly size records for network congestion and/orwait for underlying TCP retransmissions (rather than timing out).

Similarly, consider the previous examples of multi-threading within thecontext of expected use devices (see e.g., FIG. 5, and relateddiscussion); the primary user space application (e.g., video coding) andadditional secondary interactive applications (e.g., remote controlinterface, headphones, and/or other interface devices) can internallynegotiate their relative priority to the user's experience. The userspace applications can appropriately adjust their priorities for thenexus (i.e., which networking threads are serviced first and/or shouldbe deprioritized). Consequently, the user space applications candeprioritize non-essential network accesses, thereby preserving enoughCPU cycles for video decode.

As a related benefit, since a software agent represents the applicationto the kernel; the agent can trust the kernel, but the kernel may or maynot trust the agent. For example, a software agent can be used by thekernel to convey network congestion information in a trusted manner tothe application; similarly, a software agent can be used by anapplication to request a higher network priority. Notably, since asoftware agent operates from user space, the agent's privilege is notpromoted to kernel level permissions. In other words, the agent does notpermit the user application to exceed its privileges (e.g., the agentcannot commandeer the network driver at the highest network priority orforce a read/write to another application's memory space without theother kernel and/or other application's consent).

Networking extensions allow the user space application to executenetworking communications functionality within the user space andinterpose a network extension between the user space application and thekernel space. As a result, the number of cross domain accesses forcomplex layering of different protocol stacks can be greatly reduced.Limiting cross domain accesses prevents context switching and allows theuser space to efficiently police its own priorities. For example,consider the previous example of a VPN session as was previouslyillustrated in FIG. 4. By keeping the TCP/IP, Internet Protocol Security(IPsec) and TLS operations within user space, the entire tunnel can beperformed within the user space, and only cross the user/kernel domainonce.

As used herein, the term “interposition” may refer to the insertion ofan entity between two or more layers. For example, an agent isinterposed between the application and the user space networking stack.Depending on the type of agent or network extension, the interpositioncan be explicit or implicit. Explicit interposition occurs where theapplication explicitly instances the agent or network extension. Forexample, the application may explicitly call a user space tunnelextension. In contrast, implicit interposition occurs where theapplication did not explicitly instance the agent or network extension.Common examples of implicit interposition occur where one user spaceapplication sniffs the traffic or filters the content of another userspace application.

As used herein, an “instance” may refer to a single copy of a softwareprogram or other software object; “instancing” and “instantiations”refers to the creation of the instance. Multiple instances of a programcan be created; e.g., copied into memory several times. Software objectinstances are instantiations of a class; for example, a first softwareagent and second software instance are each distinct instances of thesoftware agent class.

Exemplary User Space Networking Stack

Referring now to FIG. 7, one logical block diagram of an exemplary userspace networking stack 700 is depicted. As shown, the user spacenetworking stack 700 includes an application interface 702, and anoperating system interface 704. Additionally, the user space networkingstack includes one or more user space instances of TLS 706, QUIC 708,TCP 710, UDP 712, IP 714, and ESP 716. The disclosed instances arepurely illustrative, artisans of ordinary skill in the related arts willreadily appreciate that any other user space kernel extension and/orsocket functionality may be made available within the user spacenetworking stack 700.

In one exemplary embodiment, the user space networking stack 700 isinstantiated within an application user space 718. More directly, theuser space networking stack 700 is treated identically to any one ofmultiple threads 710 within the application user space 718. Each of thecoexisting threads 720 has access to the various functions and librariesoffered by the user space networking stack via a direct function call.

As a brief aside, each of the threads 720 reside within the same addressspace. By virtue of their shared addressability, each of the threads maygrant or deny access to their portions of shared address space viaexisting user space memory management schemes and/or virtual machinetype protections. Additionally, threads can freely transfer datastructures from one to the other, without e.g., incurring cross domainpenalties. For example, TCP data 710 can be freely passed to TLS 706 asa data structure within a user space function call.

As previously noted, the user space networking stack 700 may grant ordeny access to other coexistent user space threads; e.g., a user spacethread is restricted to the specific function calls and privileges madeavailable via the application interface 702. Furthermore, the user spacenetworking stack 700 is further restricted to interfacing the operatingsystem via the specific kernel function calls and privileges madeavailable via the operating system interface 704. In this manner, boththe threads and the user space networking stack have access andvisibility into the kernel space, without compromising the kernel'ssecurity and stability.

One significant benefit of the user space networking stack 700 is thatnetworking function calls can be made without acquiring various locksthat are present in the in-kernel networking stack. As previously noted,the “locking” mechanism is used by the kernel to enforce access limitson multiple threads from multiple different user space applications;however in the user space, access to shared resources are handled withinthe context of only one user application space at a time, consequentlyaccess to shared resources are inherently handled by the singlethreading nature of user space execution. More directly, only one threadcan access the user space networking stack 700 at a time; consequently,kernel locking is entirely obviated by the user space networking stack.

Another benefit of user space network stack operation is cross platformcompatibility. For example, certain types of applications (e.g.,iTunes®, Apple Music® developed by the Assignee hereof) are deployedover a variety of different operating systems. Similarly, some emergingtransport protocols (e.g. QUIC) are ideally served by portable andcommon software between the client and server endpoints. Consistency inthe user space software implementation allows for better and moreconsistent user experience, improves statistical data gathering andanalysis, and provides a foundation for enhancing, experimenting anddeveloping network technologies used across such services. In otherwords, a consistent user space networking stack can be deployed over anyoperating system platform without regard for the native operating systemstack (e.g., which may vary widely).

Another important advantage of the exemplary user space networking stackis the flexibility to extend and improve the core protocolfunctionalities, and thus deliver specialized stacks based on theapplication's requirements. For example, a video conferencingapplication (e.g., Face Time® developed by the Assignee hereof) maybenefit from a networking stack catered to optimize performance forreal-time voice and video-streaming traffics (e.g., by allocating moreCPU cycles for video rendering, or conversely deprioritizing unimportantancillary tasks). In one such variant, a specialized stack can bedeployed entirely within the user space application, without specializedkernel extensions or changes to the kernel. In this manner, thespecialized user space networking stack can be isolated from networkingstacks. This is important both from a reliability standpoint (e.g.,updated software doesn't affect other software), as well as to minimizedebugging and reduce development and test cycle times.

Furthermore, having the network transport layer (e.g. TCP, QUIC) residein user space can open up many possibilities for improving performance.For example, as previously alluded to, applications (such as TLS) can bemodified depending on the underlying network connections. User spaceapplications can be collapsed or tightly integrated into networktransports. In some variants, data structure sizes can be adjusted basedon immediate lower layer network condition information (e.g., toaccommodate or compensate for poor network conditions). Similarly,overly conservative or under conservative transport mechanisms can beavoided (e.g., too much or not enough buffering previously present atthe socket layer). Furthermore, unnecessary data copies and/ortransforms can be eliminated and protocol signaling (congestion, error,etc.) can be delivered more efficiently.

In yet another embodiment, the exemplary user space networking stackfurther provides a framework for both networking clients and networkingproviders. In one such variant, the networking client framework allowsthe client to interoperate with any network provider (including thelegacy BSD stack). In one such variant, the network provider frameworkprovides consistent methods of discovery, connection, and data transferto networking clients. By providing consistent frameworks for clientsand providers which operate seamlessly over a range of differenttechnologies (such as a VPN, Bluetooth, Wi-Fi, cellular, etc.), theclient software can be greatly simplified while retaining compatibilitywith many different technologies.

Packet-Based Processing and Port-Batching

FIG. 8 is a logical flow diagram of packet-based processing operationswithin a nexus, useful to illustrate various aspects of the presentdisclosure. While the following discussion is presented in the contextof packet reception, analogous behavior is present in packettransmission.

As shown in the illustrated example, the nexus 800 dequeues receivedpackets from the driver packet pool ring (step 802). For each packet,the nexus inspects the packet header to extract flow information (step804). At step 806, the flow information is used to lookup a destinationpacket pool from a flow table. Thereafter the packet is enqueued to itscorresponding destination packet pool (step 808). After all the packetshave been processed, the nexus can notify destination user spaceapplications of packet delivery. Thereafter, the user space applicationsmay receive their pool of packets. The aforementioned packet-basedprocessing is a substantial improvement over traditional BSD techniquesbecause packets are routed without substantial protocol processing(e.g., the higher protocol layers (TCP, IP, etc.) are handled as neededby the user space application.)

FIG. 9 is a logical block diagram of packet-based routing within a userspace networking stack architecture, useful to illustrate variousaspects of the present disclosure. In FIG. 9, a network interface card(NIC) driver 902 receives packets via ongoing network communications.Packets are locally buffered within the NIC's packet pool 904 until theyare ready for release. Notably, the NIC driver 902 does not havevisibility into higher layers of the protocol stack (e.g., source anddestination addresses), thus packets are typically stored as they arereceived. The kernel space nexus 906 dequeues each packet from thedriver packet pool 904 and processes packets one at a time; thereafterthe packets are enqueued into their associated user space pool 908A,908B. The user space applications 910A, 910B can retrieve theircorresponding packet pools on an as-needed basis.

FIG. 10 is a graphical representation of memory allocations for packetdata structures, useful to explain various aspects of the presentdisclosure. As shown in FIG. 10, packet pool memory allocations aresplit into a number of segments 1002 (e.g., 32 KB), each segment iscomposed of buffers 1004 (e.g., 2 KB) which hold packets andcorresponding metadata 1006 (where available). The aforementionedbatched packet-based routing copies packets from the driver packet pool1010 to corresponding packet pools (1020, 1030); this entails distinctread and write (read-write) accesses for each buffer 1004 and itscorresponding metadata 1006.

As an important observation, the driver writes data packets into thebuffers of its packet pool according to its own operationalconsiderations (which may be out-of-sequence); the driver relies onhigher layers of the network stack (e.g., the TCP/IP layers) torearrange packets into their correct sequence. Notably, traditionalnetwork devices use a single IP address for all network connectionsbecause all user space applications are serviced by a single monolithicnetwork stack. Packet data is written to the aforementioned sockets fordelivery to endpoint applications. Certain “smart” NIC drivers mayadditionally tag data packets with metadata that is useful forpacket-based processing. As but one such example, so-called LargeReceive Offload (LRO) and/or Large Send Offload (LSO) capable NICs mayprovide source and/or destination port information in packet metadata.By convention, certain network ports require special handling; thus, LROand LSO may also batch packet data according to ports. So-called“port-batching” is used within high-end network routers and switches toprioritize/de-prioritize port specific tasks (e.g., HTTP connectionestablishment via port 80, etc.)

Within the context of user space networking architectures, hardwareacceleration (e.g., port-batching) is often suboptimal because itassumes the presence of a monolithic kernel space network stack. Forexample, since the aforementioned user space networking architecturesupports a distinct user space network stack for each application, portidentifiers alone are not sufficient to identify the destination port(concurrently running stacks may have conflicting usages of the sameport identifiers.) Consequently, port-batching does not significantlyimprove packet-based routing and, in fact, incorrectly orders packetswithin user packet pools.

For example, each packet is sequentially dequeued from the driver pooland then enqueued into its corresponding user space packet pools by thenexus (kernel space). However, port-batching packets at the driver poolresults in out-of-sequence delivery to user space packet pools. As aresult, user applications must also parse each packet again to arrangethe packets into their correct sequence. Furthermore, priority handlingassociated with port-batching may inadvertently cause the early or latedelivery of collateral packets. For example, a prioritized port packetthat must be immediately delivered to a user space application will alsocause the early delivery of all of the packets within the application'spool (even though the pool may have incomplete packet flows). In otherwords, the port-batched packet-based routing mechanism causesinefficient downstream processing.

Even though the aforementioned user space networking architecturesubstantially improves over traditional networking architectures, packetrouting continues to be a limiting factor in overall device performance.

Flow-Batching and Flow-Based Processing

In one exemplary embodiment, the nexus classifies and batches packetflows into flow data structures (“flow-batching”) for flow-basedprocessing. Specifically, the nexus transfers flow-batched datastructures (rather than individual packets) between its user and driverspace applications. The nexus may additionally leverage user and/ordriver space metadata to manage flow-batched data structures.Flow-batching enables in-kernel flow-based processing for packet flows.As a related benefit, flow-based processing improves instructionlocality; e.g., hardware accelerators can create metadata to tag flows,the kernel can organize flows based on hardware metadata and kernelinformation, and applications can leverage flow information based onuser space considerations. The exemplary processes described hereinpreserve information for downstream processing which may help reduceprocessing costs.

FIG. 11A is a logical block diagram of flow-based routing process 1120within an exemplary nexus, in accordance with the various principlesdescribed herein. Initially, the nexus 1100, which can be implemented byprocessor 1103, dequeues packets from the driver pool. For each packetof the driver pool (step 1102), information is extracted to generate a“flow key” (step 1104). In one exemplary embodiment, the flow key mayinclude protocol information (e.g., IP version information, IP protocol,etc.) and source and destination information (e.g., source anddestination IP addresses, and source and destination port identifiers).The nexus uses the flow key to lookup existing flows from a flow table1115 stored in a storage device 1101 (step 1106). In some embodiments,storage device 1101 can be main memory or CPU cache. If a packet has thesame flow key as an existing flow, then the packet is batched (e.g.stored) with the other packets of the flow (step 1108); if no matchingflow key is found, then a new flow key entry is added to the flow table1115. Notably, flow keys are unique to each network flow, and aredifferentiable between different user space networking stacks. Packetsmay still be received out-of-sequence due to delivery errors, etc.however batching packets into flows (flow-batching) prevents inadvertentintermingling of packet flows.

Additionally, as depicted in step 1102, a flow may not be released bythe nexus until a release condition exists (e.g., all the packets havearrived, a port event specific to the flow has occurred, etc.) In otherwords, packet-based processing of a driver pool processes all thepackets of the driver pool for delivery; in contrast, flow-basedprocessing delivers flows according to flow-specific considerations.

In the illustrated embodiment, the nexus 1100 may additionally imposeflow control on flows that are ready to be delivered. Generally, flowcontrol refers to the process of managing data transfer rates betweentwo nodes; e.g., to prevent a fast sender from overwhelming a slowrecipient, or a slow sender from reserving too many resources of a fastrecipient. For example, a completed flow may be held at step 1110 untilmore flows are ready for delivery. Other in-kernel flow control may beperformed when necessary (e.g., prioritization, deprioritization,aggregation, status reporting, feedback, error detection, errorrecovery, retransmission, etc.) Thereafter, the flow data structures areenqueued for delivery to their corresponding pools (step 1114).

FIG. 11B is a flow diagram of a process 1150 for flow-based routingwithin an exemplary nexus, in accordance with the various principlesdescribed herein. Process 1150 can be performed by processor 1103coupled to storage device 1101, where flow table 1115 is stored.

At 1151, processor 1103 receives a packet from a packet pool. The packetis associated with one or more flows, where the one or more flows areindividually identified by corresponding one or more flow keys. At 1153,processor 1103 extracts information from the packet to generate a flowkey of the packet indicating a flow for the packet. In aspects, flowkeys are unique (e. g., individual) to each of the network flows. Hence,a first flow is identified by a first flow key, and a second flow isidentified by a second flow key different from the first flow key. At1154, processor 1103 looks up the flow key in flow table 1115 in storagedevice 1101 to determine whether there is an existing flow key stored inthe flow table matching the flow key of the packet.

At 1155, responsive to the existing flow key matching the flow key ofthe packet, processor 1103 stores the packet into a batch of packets ofthe existing flow. The batch of packets of the existing flow are storedin a flow-batched data structure. At 1159, processor 1103 releases,responsive to a release condition being met, the batch of packets of theexisting flow to a user space protocol stack within a user spaceapplication through a channel communicatively coupled to the nexus andthe user space protocol stack.

In addition, at 1156, responsive to no existing flow key stored in theflow table 1115 matching the flow key of the packet, processor 1103 mayadd the flow key of the packet to flow table 1115. At 1157, processor1103 may generate a flow-batched data structure of the particular flowidentified by the flow key of the packet. At 1158, processor 1103 maystore the packet into the flow-batched data structure of the particularflow identified by the flow key of the packet. Similarly, at 1159,processor 1103 can release, responsive to a release condition being met,the batch of packets of the existing flow to a user space protocol stackwithin a user space application through a channel communicativelycoupled to the nexus and the user space protocol stack.

FIG. 12 is a logical flow diagram of flow-based processing within anexemplary nexus, in accordance with the various principles describedherein. The network interface card (NIC) driver 1202 receives andbuffers packets; e.g., packets are buffered in packet pool 1204 as theyare received. In some embodiments, a smart NIC may inspect the packetsand attach useful metadata (e.g., port information, etc.)

The exemplary kernel space nexus 1206 reads each packet from the driverpacket pool 1204, and batches packet flows based on a flow key that isunique to the network connection (e.g., source and destination IPaddresses, and source and destination port identifiers.) In oneembodiment, the flow-batched data structures may further leveragemetadata provided from e.g., applications, drivers, etc. For instance,the flow-batched data structures may use port metadata provided byLRO/LSO capable NIC hardware.

In one embodiment, flow-batched data structures may represent flows ofpacket data to facilitate downstream processing. For example, as shownin FIG. 12, four (4) packets of the driver packet pool (d₀, d₁, d₃, d₅)may be batched into a first flow-batched data structure 1212 (referredto as a “chain of packets”) and a second flow-batched data structure1214 (referred to as a “super packet”). In one exemplary embodiment, a“chain” of packets 1212 refers to a linked list of data packets that arespecific to a flow. The chain may be routed and/or released togetherthereby facilitating flow-based routing. However, the chain of packets1212 preserves each packet's individual metadata; packets may be addedor removed from the chain. In other words, each packet of the chain ofpackets retains its autonomy (an individual packet may be corrupted andACK'd, etc.) In contrast, the exemplary super packet 1214 merges packetmetadata such that all packets share the same metadata; super packets1214 reduce overall size and simplify handling, however a super packet1214 must be handled in aggregate. Notably, super packets 1214 may beconverted into chains of packets 1212 merely by duplicating redundantmetadata and vice versa.

In one exemplary embodiment, once the nexus has batched packets from thedriver pool into distinct flow-batched data structures, each flow may beseparately held or released in the packet pools 1208A, 1208B withoutimpacting other flows. For example, low-latency flows may be released assoon as packets arrive to ensure timely delivery; however, flows forbulk data transfers may be queued to maximize throughput. Additionally,the nexus can leverage driver metadata to organize and/or manage flows;thus, the valuable insights provided by hardware may be leveraged by thenexus (and preserved for downstream processing at user spaceapplications as well).

FIG. 13 is a graphical representation of memory allocations forexemplary packet, super packet, and chained packet data structures, inaccordance with the various principles described herein. As shown inFIG. 13, driver pool memory allocations may be divided in a similarmanner to the packet pool memory allocations of FIG. 10; for example,each segment 1302 (e.g., 32 KB) is composed of buffers 1304 (e.g., 2 KB)which hold packets and corresponding metadata 1306. In contrast,flow-batched data structures may be stored so as to align with nativememory boundaries of the operating system (segments, buffers, etc.)Natively aligned flow-batched data structures facilitate non-blockingflow access; in other words, access to the packets of one flow-batcheddata structure is not dependent on (and is not blocked by) other datastructures.

For example, packets for flow A and flow B are aligned on separatesegments 1312 and buffers 1314; each flow has different correspondingmetadata 1316. Flow C is a chain of packets which aligns eachconstituent packet on buffer boundaries. For instance, the first packetis aligned to the first buffer 1324 of the segment 1322. The secondpacket is aligned with the second buffer 1324 of the segment 1322. Bothpackets have their own metadata 1326. Flow D is a super packet thatcompresses metadata to reduce data size; only the first packet aligns tosegment boundaries 1332. Subsequent packets may or may not align tobuffer boundaries 1334. Only one copy of metadata 1336 is kept (thesecond packet references a pointer to the first packet's metadata).

Various embodiments can be implemented, for example, using one or morecomputer systems, such as computer system 1400 shown in FIG. 14.Computer system 1400 can be used, for example, to implement processes ofFIGS. 11A-11B. For example, computer system 1400 can implement andexecute a set of instructions comprising operations for managing a userspace networking stack, and flow-based processing and routing within auser space networking stack architecture as illustrated in FIGS. 1-13.Computer system 1400 can be any computer capable of performing thefunctions described herein.

Computer system 1400 includes one or more processors (also calledcentral processing units, or CPUs), such as a processor 1404. Processor1404 is connected to communication infrastructure or bus 1406.

One or more processors 1404 may each be a graphics processing unit(GPU). In an embodiment, a GPU is a processor that is a specializedelectronic circuit designed to process mathematically intensiveapplications. The GPU may have a parallel structure that is efficientfor parallel processing of large blocks of data, such as mathematicallyintensive data common to computer graphics applications, images, videos,etc.

Computer system 1400 also includes user input/output device(s) 1403,such as monitors, keyboards, pointing devices, etc., that communicatewith communication infrastructure 1406 through user input/outputinterface(s) 1402.

Computer system 1400 also includes a main or primary memory 1408, suchas random access memory (RAM). Main memory 1408 may include one or morelevels of cache. Main memory 1408 has stored therein control logic(i.e., computer software) and/or data.

Computer system 1400 may also include one or more secondary storagedevices or memory 1410. Secondary memory 1410 may include, for example,a hard disk drive 1412 and/or a removable storage device or drive 1414.Removable storage drive 1414 may be a floppy disk drive, a magnetic tapedrive, a compact disk drive, an optical storage device, tape backupdevice, and/or any other storage device/drive.

Removable storage drive 1414 may interact with a removable storage unit1418. Removable storage unit 1418 includes a computer usable or readablestorage device having stored thereon computer software (control logic)and/or data. Removable storage unit 1418 may be a floppy disk, magnetictape, compact disk, DVD, optical storage disk, and/any other computerdata storage device. Removable storage drive 1414 reads from and/orwrites to removable storage unit 1418 in a well-known manner.

According to an exemplary embodiment, secondary memory 1410 may includeother means, instrumentalities or other approaches for allowing computerprograms and/or other instructions and/or data to be accessed bycomputer system 1400. Such means, instrumentalities or other approachesmay include, for example, a removable storage unit 1422 and an interface1420. Examples of the removable storage unit 1422 and the interface 1420may include a program cartridge and cartridge interface (such as thatfound in video game devices), a removable memory chip (such as an EPROMor PROM) and associated socket, a memory stick and USB port, a memorycard and associated memory card slot, and/or any other removable storageunit and associated interface.

Computer system 1400 may further include a communication or networkinterface 1424. Communication interface 1424 enables computer system1400 to communicate and interact with any combination of remote devices,remote networks, remote entities, etc. (individually and collectivelyreferenced by reference number 1428). For example, communicationinterface 1424 may allow computer system 1400 to communicate with remotedevices 1428 over communications path 1426, which may be wired and/orwireless, and which may include any combination of LANs, WANs, theInternet, etc. Control logic and/or data may be transmitted to and fromcomputer system 1400 via communication path 1426.

In an embodiment, a tangible, non-transitory apparatus or article ofmanufacture comprising a tangible, non-transitory computer useable orreadable medium having control logic (software) stored thereon is alsoreferred to herein as a computer program product or program storagedevice. This includes, but is not limited to, computer system 1400, mainmemory 1408, secondary memory 1410, and removable storage units 1418 and1422, as well as tangible articles of manufacture embodying anycombination of the foregoing. Such control logic, when executed by oneor more data processing devices (such as computer system 1400), causessuch data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparentto persons skilled in the relevant art(s) how to make and useembodiments of this disclosure using data processing devices, computersystems and/or computer architectures other than that shown in FIG. 14.In particular, embodiments can operate with software, hardware, and/oroperating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and notany other section, is intended to be used to interpret the claims. Othersections can set forth one or more but not all exemplary embodiments ascontemplated by the inventor(s), and thus, are not intended to limitthis disclosure or the appended claims in any way.

While this disclosure describes exemplary embodiments for exemplaryfields and applications, it should be understood that the disclosure isnot limited thereto. Other embodiments and modifications thereto arepossible, and are within the scope and spirit of this disclosure. Forexample, and without limiting the generality of this paragraph,embodiments are not limited to the software, hardware, firmware, and/orentities illustrated in the figures and/or described herein. Further,embodiments (whether or not explicitly described herein) havesignificant utility to fields and applications beyond the examplesdescribed herein.

Embodiments have been described herein with the aid of functionalbuilding blocks illustrating the implementation of specified functionsand relationships thereof. The boundaries of these functional buildingblocks have been arbitrarily defined herein for the convenience of thedescription. Alternate boundaries can be defined as long as thespecified functions and relationships (or equivalents thereof) areappropriately performed. Also, alternative embodiments can performfunctional blocks, steps, operations, methods, etc. using orderingsdifferent than those described herein.

References herein to “one embodiment,” “an embodiment,” “an exampleembodiment,” or similar phrases, indicate that the embodiment describedcan include a particular feature, structure, or characteristic, butevery embodiment can not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it would be within the knowledge of persons skilled in therelevant art(s) to incorporate such feature, structure, orcharacteristic into other embodiments whether or not explicitlymentioned or described herein. Additionally, some embodiments can bedescribed using the expression “coupled” and “connected” along withtheir derivatives. These terms are not necessarily intended as synonymsfor each other. For example, some embodiments can be described using theterms “connected” and/or “coupled” to indicate that two or more elementsare in direct physical or electrical contact with each other. The term“coupled,” however, can also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other.

As used herein, the term “native” and “natural” refer to memoryboundaries of the operating system that are addressable via the memorymanagement unit (MMU). Native memory boundaries of the operating systemcan be exposed to user space applications directly; the kernel does notneed to parse network protocol data structures using the aforementionedread-write accesses. In some variants, transfers based on native memoryboundaries enable “zero-copy” data transfers (where the processor doesnot copy data from one memory location to another memory location).Additionally, since each flow data structure is isolated from other flowdata structures, flows do not block one another. Thus, flows may bereleased when they are ready rather than when e.g., a segment is fullypacked with packets. More directly, the exemplary flow data structureprioritizes ease of access over compactness.

It will be recognized that while certain embodiments of the presentdisclosure are described in terms of a specific sequence of steps of amethod, these descriptions are only illustrative of the broader methodsdescribed herein, and may be modified as required by the particularapplication. Certain steps may be rendered unnecessary or optional undercertain circumstances. Additionally, certain steps or functionality maybe added to the disclosed embodiments, or the order of performance oftwo or more steps permuted. All such variations are considered to beencompassed within the disclosure and claimed herein.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it will beunderstood that various omissions, substitutions, and changes in theform and details of the device or process illustrated may be made bythose skilled in the art without departing from principles describedherein. The foregoing description is of the best mode presentlycontemplated. This description is in no way meant to be limiting, butrather should be taken as illustrative of the general principlesdescribed herein. The scope of the disclosure should be determined withreference to the claims.

What is claimed is:
 1. A method for managing a user space protocolstack, the method comprising: receiving, by a nexus in a kernel space, apacket from a packet pool, wherein the packet is associated with one ormore flows, wherein the one or more flows are individually identified bycorresponding one or more flow keys; extracting information from thepacket to generate a flow key of the packet indicating a particular flowfor the packet; looking up the flow key in a flow table to determinewhether there is an existing flow key stored in the flow table matchingthe flow key of the packet; storing, responsive to the existing flow keymatching the flow key of the packet, the packet into a batch of packetsof the existing flow, wherein the batch of packets of the existing floware stored in a flow-batched data structure; and releasing, responsiveto a release condition being met, the batch of packets of the existingflow to a user space protocol stack within a user space applicationthrough a channel communicatively coupled to the nexus and the userspace protocol stack.
 2. The method of claim 1, further comprising:adding, responsive to no existing flow key stored in the flow tablematching the flow key of the packet, the flow key of the packet to theflow table; generating a flow-batched data structure of the particularflow identified by the flow key of the packet; and storing the packetinto the flow-batched data structure of the particular flow identifiedby the flow key of the packet.
 3. The method of claim 1, furthercomprising: imposing a flow control on the released the batch of packetsof the existing flow to manage a data transfer rate of the existingflow.
 4. The method of claim 1, wherein the packet pool includes packetsreceived by a network interface card (NIC) driver.
 5. The method ofclaim 1, wherein the release condition includes an indication that allpackets of the existing flow have arrived in the nexus, or apredetermined port event associated with the existing flow has occurred.6. The method of claim 1, wherein the flow key includes protocolinformation, IP version information, source information for the flow,destination information for the flow, a source IP address, a destinationIP addresses, a source port identifier, or a destination portidentifier. The method of claim 1, wherein the flow-batched datastructure includes a metadata generated by hardware, or a metadataprovided by the user space application.
 8. The method of claim 7,wherein the first flow includes a first metadata, and a second flowincludes a second metadata different from the first metadata.
 9. Themethod of claim 1, wherein the flow-batched data structure includes achain of packets or a super packet.
 10. The method of claim 1, whereinthe batch of packets stored in the flow-batched data structure of theexisting flow is a first batch of packets stored in a first flow-batcheddata structure of a first existing flow; and the method furthercomprises: storing one or more packets into a second batch of packetsstored in a second flow-batched data structure of a second existingflow; and releasing the first batch of packets stored in the firstflow-batched data structure of the first existing flow without impactingthe second existing flow.
 11. The method of claim 1, wherein the packetof the packet pool is stored in one or more buffers of a networkinterface card (NIC), and the flow-batched data structure is stored inmemory aligned with native memory boundaries of the operating system.12. A system configured for managing packet flows, the systemcomprising: a storage device to store a flow table; and a processorcoupled to the storage device, and configured to operate a nexus in akernel space, wherein the nexus is configured to: receive a packet froma packet pool, wherein the packet is associated with one or more flows,wherein the one or more flows are individually identified bycorresponding one or more flow keys; extract information from the packetto generate a flow key of the packet indicating a particular flow forthe packet; look up the flow key in the flow table in the storage todetermine whether there is an existing flow key stored in the flow tablematching the flow key of the packet; store, responsive to the existingflow key matching the flow key of the packet, the packet into a batch ofpackets of the existing flow, wherein the batch of packets of theexisting flow are stored in a flow-batched data structure; and release,responsive to a release condition being met, the batch of packets of theexisting flow to a user space protocol stack within a user spaceapplication through a channel communicatively coupled to the nexus andthe user space protocol stack.
 13. The system of claim 12, wherein thenexus is further configured to: add, responsive to no existing flow keystored in the flow table matching the flow key of the packet, the flowkey of the packet to the flow table; generate a flow-batched datastructure of the particular identified by the flow key of the packet;and store the packet into the flow-batched data structure of theparticular flow identified by the flow key of the packet.
 14. The systemof claim 12, wherein the nexus is further configured to: impose a flowcontrol on the released the batch of packets of the existing flow tomanage a data transfer rate of the existing flow.
 15. The system ofclaim 12, wherein the packet pool includes packets received by a networkinterface card (NIC) driver.
 16. The system of claim 12, wherein theflow key includes protocol information, IP version information, sourceinformation for the flow, destination information for the flow, a sourceIP address, a destination IP addresses, a source port identifier, or adestination port identifier.
 17. The system of claim 12, wherein theflow-batched data structure includes a metadata generated by hardware,or a metadata provided by the user space application.
 18. The system ofclaim 12, wherein the flow-batched data structure includes a chain ofpackets or a super packet.
 19. A non-transitory computer readable mediumcomprising computer programs stored thereon, when executed by aprocessor, cause a computerized apparatus to: receive, by a nexus in akernel space, a packet from a packet pool, wherein the packet isassociated with one or more flows, wherein the one or more flows areindividually identified by corresponding one or more flow keys; extractinformation from the packet to generate a flow key of the packetindicating a particular flow for the packet; look up the flow table in astorage, based on the flow key, to determine whether there is anexisting flow key stored in the flow table matching the flow key of thepacket; store, responsive to the existing flow key matching the flow keyof the packet, the packet into a batch of packets of the existing flow,wherein the batch of packets of the existing flow are stored in aflow-batched data structure; and release, responsive to a releasecondition is met, the batch of packets of the existing flow to a userspace protocol stack within a user space application through a channelcommunicatively coupled to the nexus and the user space protocol stack.20. The non-transitory computer readable medium of claim 19, wherein thepacket pool includes packets received by a network interface card (NIC)driver; wherein the flow-batched data structure includes a metadatagenerated by hardware, or a metadata provided by the user spaceapplication; and wherein the flow-batched data structure includes achain of packets or a super packet.